Performing operations using a mirror in an artificial reality environment

ABSTRACT

This disclosure describes an artificial reality system that presents artificial reality content in the context of a physical environment that includes a mirror or other reflective surface. In one example, this disclosure describes a method that includes capturing capture data representative of a physical environment, wherein the physical environment includes a reflective surface and a plurality of objects, determining a pose of the HMD, determining a map of the physical environment, wherein the map includes position information about the reflective surface and position information about each of the plurality of physical objects in the physical environment, identifying a visible object from among the plurality of physical objects, and generating artificial reality content associated with the visible object.

CROSS REFERENCE

This application is a continuation application of and claims priority toU.S. patent application Ser. No. 16/454,438 filed on Jun. 27, 2019,which is hereby incorporated by reference herein in its entirety.

TECHNICAL FIELD

This disclosure generally relates to artificial reality systems, such asvirtual reality, mixed reality and/or augmented reality systems, andmore particularly, to presentation of content and performing operationsin artificial reality applications.

BACKGROUND

Artificial reality systems are becoming increasingly ubiquitous withapplications in many fields such as computer gaming, health and safety,industrial, and education. As a few examples, artificial reality systemsare being incorporated into mobile devices, gaming consoles, personalcomputers, movie theaters, and theme parks. In general, artificialreality is a form of reality that has been adjusted in some mannerbefore presentation to a user, which may include, e.g., a virtualreality (VR), an augmented reality (AR), a mixed reality (MR), a hybridreality, or some combination and/or derivatives thereof.

Typical artificial reality systems include one or more devices forrendering and displaying content to users. As one example, an artificialreality system may incorporate a head-mounted display (HMD) worn by auser and configured to output artificial reality content to the user.The artificial reality content may include a number of different typesof artificial reality content, including see-through AR, overlay AR,completely-generated content, generated content combined with capturedcontent (e.g., real-world video and/or images), or other types. Duringoperation, the user typically interacts with the artificial realitysystem to select content, launch applications or otherwise configure thesystem.

SUMMARY

This disclosure describes an artificial reality system that presentsartificial reality content in the context of a physical environment thatincludes a mirror or other reflective surface. Techniques describedherein include identifying physical objects within a physicalenvironment that have reflections that are visible within a physicalmirror, and presenting, within an HMD, artificial reality contentoverlaid on the reflections of those physical objects within the mirror.

In some examples, various types of virtual content may be presentedthrough a mirror (including hats, other articles of clothing, and viewsof the user's room reflected back through the mirror), and such contentmay be positioned relative to a physical object in the room, or in somecases, “locked” to a moving user or to a body part (e.g., head, arm,foot) of a moving user. Touch interactions with the mirror may be usedto for gating or interacting with a user interface menu. In response tosuch interactions, the artificial reality content presented as anoverlay to reflections in the mirror may be modified or updatedappropriately. In general, the mirror (or the area near the mirror) maybe used to trigger, gate, or enable performance of certain operations,including presentation of a user interface (UI) to customize optionsrelating to presented artificial reality content. For example, the UImight be used to select overlaid artificial reality content, enablingusers to “try on” clothing or accessories or exchange such clothing oraccessories with other users.

Techniques are also described for use of a physical mirror to performspecific computing operations. For instance, teleconferencing operationsare described in which a user might look into a mirror and see bothhimself or herself and the remote person participating in theteleconference. The remote person may be presented as an avatar,presented normally, or presented as an image having overlaid virtualapparel (e.g., a hat). Where multiple users participate in the call, allusers might simultaneously be visible within a mirror located withineach user's physical environment. Touch interactions may enable exchangeof digital content with other teleconference participants, or with otherremote systems. As described herein, a mirror may serve as an intuitivepoint of reference for gating or triggering various operations involvingmultiple users.

In addition, techniques are described in which an artificial realitysystem uses a mirror for movement instruction, such as dance or exerciselessons, or for other instructions, such as those describing how to usea new product. In some examples, an artificial reality system may detecta series of movements made by a user, and compare those movements to amodel set of movements. Based on the comparison, the artificial realitysystem may determine whether a user's movements might be improved ormodified, and if so, artificial reality content may be presented toillustrate movements that more appropriately align with the model set ofmovements.

In some examples, this disclosure describes operations performed by anartificial reality system in accordance with one or more aspects of thisdisclosure. In one specific example, this disclosure describes a systemcomprising an image capture system configured to capture image datarepresentative of a physical environment that includes a reflectivesurface and a plurality of objects, wherein the image capture systemincludes a head-mounted display (HMD) worn by a first user; a mappingengine configured to determine, based on the image data, a map of thephysical environment including position information about the HMD, thereflective surface, and each of the plurality of physical objects; andan application engine configured to: enable communication over a networkbetween the first user and a second user that is in a second physicalenvironment, determine, based on the position of the HMD and thereflective surface, whether the first user is located in front of thereflective surface, responsive to determining that the first user islocated in front of the reflective surface, output, over the network, aplurality of images of the first user, and responsive to determiningthat the first user is not located in front of the reflective surface,refrain from outputting images of the first user.

In another example, this disclosure describes a method capturing, by acomputing system that includes a head-mounted display (HMD) worn by afirst user, image data representative of a physical environment thatincludes a reflective surface, a plurality of objects, and the HMD;determining, by the computing system and based on the image data, a mapof the physical environment including position information about theHMD, the reflective surface, and each of the plurality of physicalobjects; enabling, by the computing system, communication over a networkbetween the first user and a second user, wherein the second user islocated in a second physical environment, determining, by the computingsystem and based on the position of the HMD and the reflective surface,whether the first user is located in front of the reflective surface;responsive to determining that the first user is located in front of thereflective surface, outputting, by the computing system and over thenetwork, a plurality of images of the first user; and responsive todetermining that the first user is not located in front of thereflective surface, refraining from outputting images of the first user.

In another example, this disclosure describes a non-transitorycomputer-readable medium comprising instructions for causing processingcircuitry of an artificial reality system having an image capture systemthat includes a head-mounted display (HMD) worn by a first user toperform operations comprising: capturing image data representative of aphysical environment that includes a reflective surface, a plurality ofobjects, and the HMD; determining, based on the image data, a map of thephysical environment including position information about the HMD, thereflective surface, and each of the plurality of physical objects;enabling communication over a network between the first user and asecond user, wherein the second user is located in a second physicalenvironment; determining, based on the position of the HMD and thereflective surface, whether the first user is located in front of thereflective surface; responsive to determining that the first user islocated in front of the reflective surface, outputting, over thenetwork, a plurality of images of the first user; and responsive todetermining that the first user is not located in front of thereflective surface, refraining from outputting images of the first user.

The details of one or more examples of the techniques of this disclosureare set forth in the accompanying drawings and the description below.Other features, objects, and advantages of the techniques will beapparent from the description and drawings, and from the claims.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1A is a conceptual diagram illustrating an example artificialreality system that generates artificial reality content associated withimages reflected by a mirror, in accordance with one or more aspects ofthe present disclosure.

FIG. 1B is a conceptual diagram illustrating an example artificialreality system that generates a user interface when presentingartificial reality content associated with images reflected by a mirror,in accordance with one or more aspects of the present disclosure.

FIG. 1C is a conceptual diagram illustrating an example artificialreality system that generates artificial reality content associated withimages of another user reflected by a mirror, in accordance with one ormore aspects of the present disclosure.

FIG. 2 is an illustration depicting an example head-mounted displayconfigured to operate in accordance with the techniques of thedisclosure.

FIG. 3 is a block diagram showing example implementations of an exampleconsole and an example HMD, in accordance with one or more aspects ofthe present disclosure.

FIG. 4 is a block diagram depicting an example of a user device for anartificial reality system, in accordance with one or more aspects of thepresent disclosure.

FIG. 5A and FIG. 5B are example diagrams illustrating one possibletechnique for identifying physical objects that are reflected in amirror or other reflective surface, in accordance with one or moreaspects of the present disclosure.

FIG. 6 is an example diagram illustrating another possible technique foridentifying physical objects that are reflected in a mirror or otherreflective surface, in accordance with one or more aspects of thepresent disclosure.

FIG. 7A, FIG. 7B, and FIG. 7C are conceptual diagrams illustratingexample techniques for enabling multiple users to communicate and/orperform other operations using a mirror, in accordance with one or moreaspects of the present disclosure.

FIG. 8 is a conceptual diagram illustrating an example technique forproviding movement instruction using a mirror, in accordance with one ormore aspects of the present disclosure.

FIG. 9 is a flow diagram illustrating operations performed by an exampleartificial reality console in accordance with one or more aspects of thepresent disclosure.

DETAILED DESCRIPTION

FIG. 1A is a conceptual diagram illustrating an example artificialreality system that generates artificial reality content associated withimages reflected by a mirror, in accordance with one or more aspects ofthe present disclosure. In the example of FIG. 1A, artificial realitysystem 100A is depicted within physical environment 120A. In FIG. 1A,physical environment 120A is shown as a room that includes user 101 anda number of real world or physical objects, including HMD 112, lamp 107,mirror 109, frame 110, object 141, object 142, and object 143. User 101is positioned within physical environment 120A near mirror 109, and inthe example shown, user 101 is positioned such that he can see areflection of himself (101′) in mirror 109. Mirror 109 is mounted withinmirror frame 110. User 101 is wearing a shirt having the numeral “1” onit and having a single stripe on the right sleeve. As would be expected,that numeral “1” and stripe are shown as a reflected (reversed) image inmirror 109. Similarly, because of the relative positions of user 101,HMD 112, and object 141 to mirror 109, user 101 can also see areflection of HMD 112 (112′) and object 141 (141′) in mirror 109.Although examples described herein are principally described in terms ofimages reflected by a mirror, such examples may also be applicable toany reflective surface that tends to reflect images, such as a window orthe like.

Object 141 includes two instances of the numeral “1” printed on it—oneon a near-facing side of object 141 (shown in FIG. 1A) and another on afar-facing side of object 141. The numeral “1” on the far-facing side ofobject 141 is not directly visible in FIG. 1A, but since object 141 ispositioned so that its image is reflected by mirror 109, the reflectedimage of the “1” printed on the far-facing side of object 141 is visiblein mirror 109 (as 141′). Object 142 and object 143 also each have anumeral printed on two sides: object 142 has the numeral “2” printed ona near-facing side and a far-facing side of object 142, and object 143has the numeral “3” printed on a near-facing side and a far facing-sideof object 143. In the example of FIG. 1A, neither object 142 nor object143 are visible in mirror 109, so only the numerals printed on thenear-facing sides of object 142 and object 143 can be seen in FIG. 1A.

In FIG. 1A, artificial reality system 100A includes head-mounted display(HMD) 112, console 106, one or more sensors 190, and cameras 192A and192B (collectively “cameras 192,” representing any number of cameras).Although in some examples external sensors 190 and cameras 192 may bestationary devices (e.g., affixed to the wall), in other examples one ormore of external sensors 190 and/or cameras 192 may be included withinHMD 112, within a user device (not shown), or within any other device orsystem. As shown in FIG. 1A, HMD 112 is typically worn by user 101 andincludes an electronic display and optical assembly for presentingartificial reality content 122A to the user. In addition, HMD 112 mayinclude one or more sensors (e.g., accelerometers) for tracking motionof the HMD and may include one or more image capture devices, e.g.,cameras, line scanners and the like, for capturing image data of thesurrounding environment.

Artificial reality system 100A may use information obtained from areal-world or physical three-dimensional (3D) environment to renderartificial reality content 122A for display by HMD 112, therebypresenting the content to user 101. In the example of FIG. 1A, user 101views the artificial reality content 122A constructed and rendered by anartificial reality application executing on console 106 and/or HMD 112.As one example, artificial reality content 122 may include virtual hat123 overlaid on an image of a reflection of user reflected by mirror109. To implement such an effect, physical mirror 109 may be thought ofas having a “virtual mirror” that is coincident with mirror 109 or inthe map of the physical world at the same place as and with anorientation corresponding to an orientation of physical mirror 109.

In other examples, artificial reality content 122A may correspond tocontent rendered pursuant to a social interaction application, a videoconferencing application, a movement instruction application, analternative world application, a navigation application, an educationalapplication, gaming application, training or simulation applications,augmented reality application, virtual reality application, or othertype of applications that implement artificial reality.

In the example of FIG. 1A, console 106 is shown as a single computingdevice, such as a gaming console, workstation, a desktop computer, or alaptop. In other examples, console 106 may be distributed across aplurality of computing devices, such as a distributed computing network,a data center, or a cloud computing system. HMD 112, console 106,external sensors 190, and cameras 192, may, as shown in FIG. 1A, becommunicatively coupled via network 104, which may be a wired orwireless network, such as Wi-Fi, a mesh network or a short-rangewireless communication medium. In some examples, user 101 may use one ormore controllers (not shown) to perform gestures or other actions. Insuch an example, such controllers may be in communication with HMD 112using near-field communication or short-range wireless communicationsuch as Bluetooth, using wired communication links, or using anothertype of communication links. Although HMD 112 is shown in FIG. 1A asbeing in communication with, e.g., tethered to or in wirelesscommunication with, console 106, in some implementations HMD 112operates as a stand-alone, mobile artificial reality system. As such,some or all functionality attributed to console 106 in this disclosuremay be distributed among one or more user devices, such as one or moreinstances of HMD 112.

In some examples, an artificial reality application executing on console106 and/or HMD 112 in FIG. 1A presents artificial reality content touser 101 based on a current viewing perspective for user 101. That is,in FIG. 1A, the artificial reality application constructs artificialcontent by tracking and computing pose information for a frame ofreference for HMD 112, and uses data received from HMD 112, externalsensors 190, and/or cameras 192 to capture 3D information within thereal-word, physical 3D environment 122, such as motion by user 101and/or tracking information with respect to user 101 and one or morephysical objects, for use in computing updated pose information for acorresponding frame of reference of HMDs 112 (or another user device).As one example, the artificial reality application may render, based ona current viewing perspective determined for HMD 112, artificial realitycontent 122 having one or more artificial reality content objectsoverlaid upon images of reflected physical or real-world objects (e.g.,user 101, object 141). Further, from the perspective of HMD 112,artificial reality system 100A renders artificial reality content basedupon the estimated positions and poses for user 101 and other physicalobjects.

In the example of FIG. 1A, an in accordance with one or more aspects ofthe present disclosure, HMD 112 may present artificial reality content122A to user 101. For instance, with reference to FIG. 1A, each of HMD112, external sensors 190, and/or cameras 192 capture images withinphysical environment 120A. Console 106 receives such images anddetermines the position of physical objects within physical environment120A, including user 101, HMD 112, and mirror 109. Console 106determines the region of space reflected in mirror 109, and uses thatinformation to identify which physical objects are, from the perspectiveof HMD 112, visible in mirror 109. In some examples, determining whichphysical objects are visible in mirror 109 may involve one or more ofthe techniques described below in connection with FIG. 5A, FIG. 5B,and/or FIG. 6. Typically, console 106 does not “find” physical objectsin the room through the mirror, and as such, does not typicallyimplement a simultaneous localization and mapping (SLAM) to detect thelocation of physical objects in a reflected image, and HMD 112 is nottypically detected by the interaction with the mirror. Rather, HMD 112may determine its position in the map and communicate that position toother HMDs, if any, through console 106 or otherwise. Further, in someexamples, user 101 may define the location of mirror 109 manuallythrough input.

In the example of FIG. 1A, console 106 determines that reflected imagesof user 101 and object 141 are visible in mirror 109. Console 106generates artificial reality content 122A, and includes virtual hat 123at a position that is determined based on the location, within physicalenvironment 120A, of HMD 112. In some examples, virtual hat 123 is“locked” (or substantially locked) to the head of user 101 or to HMD 112so that when the head of user 101 or when HMD 112 moves, the appropriateposition of virtual hat 123 on the reflected image of the head of user101 would also move. In some examples, “substantially locked” may meanthat the hat 123 may move relative to head of user 101 slightly inresponse to movements of user 101, just as a physical hat might moveslightly in response to such movements. Hat 123 might also fall off ofthe head of user 101 if user 101 moved abruptly, again just as aphysical hat would fall of the head of user 101 in such a situation. Inother examples, however, “substantially locked” may encompass anypositioning of an artificial reality element that is based in some wayon the underlying physical object, and alternatively, or in addition,may encompass dynamic content that is not tightly locked but may beorganic in its movement and distance from the user.

Similarly, object 141 is presented within artificial reality content122A having an artificial reality arrow 149 next to it. Console 106causes virtual hat 123 and arrow 149 to be overlaid on images ofphysical objects captured within 100A. Console 106 causes HMD 112 topresent artificial reality content 122A to user 101 within HMD 112 inthe manner shown in FIG. 1A. Although virtual hat 123 is illustrated anddescribed as anchored or attached to the head of user 101 or to HMD 112,virtual hat 123 could be presented anywhere in the physical reflection,attached to a physical object or not attached to a physical object.Similarly, arrow 149 might be anchored to object 141 such that if object141 were moved or overturned, arrow 149 would move within artificialreality content 122A appropriately. In other examples, however, arrow149 might not be anchored to object 141, so that its position might notbe affected by movements of object 141.

Artificial reality content described herein may take any form, beyondvirtual hat 123, arrow 149, and other content presented in otherillustrations. For instance, such content may include any article ofclothing (e.g., locked to a watch, hand, shoulder, etc.) oraugmentation, and could encompass any two-dimensional orthree-dimensional static or dynamic content. Further, such artificialreality content could include masking out physical objects in themirror, so that console 106 or HMD 112 effectively removes the presenceof physical object(s) in the mirror reflection (e.g., a vampire mightnot cast a reflection in a mirror).

Further, although only one user 101 is illustrated in FIG. 1A (and insome of the other illustrations herein), multiple users, each wearing anHMD 112, may be present in a room with one (or more) mirrors. In such anexample, each of the users may be presented with augmentations similarto those described in connection with user 101. Where there are multipleHMDs, there may be a number of different ways to track physical and/orvirtual content for the HMDs. In one example, one system of sensors(e.g., sensors 190 and 192) could perform the tracking, and communicatethe results to each of the HMDs. In another example, each HMD may haveits own map and track itself, and also communicate information about itsmap and/or tracking to other HMDs. In another example, HMDs, may trackrecognized objects, and can communicate those locations to other HMDs.

FIG. 1B is a conceptual diagram illustrating an example artificialreality system that generates a user interface when presentingartificial reality content associated with images reflected by a mirror,in accordance with one or more aspects of the present disclosure. In theexample of FIG. 1B, artificial reality system 100B is depicted withinphysical environment 120B, and physical environment 120B is again shownas a room that includes user 101 and a number of physical objects. Inthe example of FIG. 1B, artificial reality system 100B includes many ofthe same elements described in connection with FIG. 1A, and elementsillustrated in FIG. 1B may correspond to elements illustrated in FIG. 1Athat are identified by like-numbered reference numerals in FIG. 1A. Ingeneral, such like-numbered elements may be implemented in a mannerconsistent with the description of the corresponding element provided inconnection with FIG. 1A or elsewhere herein, although in some examples,such elements may involve alternative implementation with more, fewer,and/or different capabilities and attributes. Accordingly, artificialreality system 100B of FIG. 1B may be described as an alternativeexample or implementation of artificial reality system 100A of FIG. 1A.

In FIG. 1B, user 101 is slightly closer to mirror 109 and is reflectedin mirror 109. Object 141 is behind user 101 and is also reflected inmirror 109. In FIG. 1B, user 101 is close enough to mirror 109 to reachmirror 109 with a hand.

In the example of FIG. 1B, and in accordance with one or more aspects ofthe present disclosure, HMD 112 may present a user interface. Forinstance, with reference to FIG. 1B, HMD 112, external sensors 190,and/or cameras 192 capture images within physical environment 120B.Console 106 determines that user 101 is physically positioned in frontof mirror 109. Responsive to determining that user 101 is positioned infront of mirror 109, console 106 generates artificial reality content122B, and includes user interface menu 124 within artificial realitycontent 122B. In the example of FIG. 1B, user interface menu 124 furtherincludes user interface elements 126. Console 106 causes artificialreality content 122B to be presented to user 101 within HMD 112. In theexample described, artificial reality content 122B may be presented touser 101 within HMD 112 simply in response to detecting that user 101 ispositioned in front of mirror 109.

In another example, HMD 112 may present a user interface in response touser input. For instance, again referring to FIG. 1B, each of HMD 112,external sensors 190, and/or cameras 192 capture images within physicalenvironment 120B. Console 106 determines that the motion and/or imagesindicate that user 101 has touched mirror 109 at touch point 119.Console 106 further determines that such a motion is a gesture thatgates or triggers presentation of a user interface. Console 106generates artificial reality content 122B and includes user interfacemenu 124 within artificial reality content 122B. Although the inputdescribed above may involve direct touching of mirror 109, in otherexamples, such input may include or may be an at-a-distance interactionwith mirror 109.

In the example of FIG. 1B, user interface menu 124 further includes userinterface elements 126. Console 106 may detect interaction with userinterface menu 124 and/or user interface elements 126 and in response,perform one or more operations. For instance, in some examples, console106 may detect that user 101 has interacted with user interface menu 124to change the appearance of virtual hat 123 (or to remove virtual hat123). In response to such interactions, console 106 may updateartificial reality content 122B appropriately. User 101 may alsointeract with user interface menu 124 to change other articles ofclothing or to customize other content being presented. In someexamples, interactions with user interface menu 124 may causemodifications to the artificial reality content presented through HMD112, which would then apply to the physical world's virtual contentbased on the chosen modifications.

In still other examples, presentation of user interface menu 124 mightnot be necessary to perform the described or other operations. Forinstance, user 101 may perform a swiping motion to change the appearanceof virtual hat 123 (e.g., to change hats), or to change other articlesof clothing that may be overlaid on images of user 101. In anotherexample, user 101 may perform gestures (e.g., hand gestures) that mayhave a particular meaning when performed in front of mirror 109 or infront of another mirror.

FIG. 1C is a conceptual diagram illustrating an example artificialreality system that generates artificial reality content associated withimages of another user reflected by a mirror, in accordance with one ormore aspects of the present disclosure. In the example of FIG. 1C,artificial reality system 100C is depicted within physical environment120C, and physical environment 120C is shown as a room having a slightlydifferent configuration or arrangement than those of FIG. 1A or FIG. 1B.In FIG. 1C, physical environment 120C includes mirror 109, but includesboth user 101 and user 102, along with object 141 and object 142. InFIG. 1C, user 101 wears HMD 112A, and user 102 wears HMD 112B, where HMD112A and HMD 112B may have similar capabilities.

However, as in FIG. 1B, artificial reality system 100C includes many ofthe same elements described in connection with FIG. 1A and FIG. 1B, andelements illustrated in FIG. 1C may correspond to elements illustratedin FIG. 1A and/or FIG. 1B that are identified by like-numbered referencenumerals. As explained in connection with FIG. 1B, such like-numberedelements may be implemented in a manner consistent with the descriptionof the corresponding element provided in connection with FIG. 1A orelsewhere herein, although in some examples, such elements may involvealternative implementation with more, fewer, and/or differentcapabilities and attributes. Accordingly, artificial reality system 100Cof FIG. 1C may be described as an alternative example or implementationof artificial reality system 100A of FIG. 1A or artificial realitysystem 100B of FIG. 1B.

In the example of FIG. 1C, and in accordance with one or more aspects ofthe present disclosure, HMD 112A may present artificial reality content122C that includes content locked to the head of another user. Forinstance, with reference to FIG. 1C, each of HMD 112A, HMD 112B,external sensors 190, and/or cameras 192 capture images within physicalenvironment 120A. Console 106 (see FIG. 1A or 1B) receives such imagesand determines the position of physical objects within physicalenvironment 120A, including user 101, HMD 112A, HMD 112B, and mirror109. Console 106 determines which physical objects are, from theperspective of HMD 112A, visible in mirror 109. In the example of FIG.1C, console 106 determines that images of user 102 and object 142 arevisible in mirror 109. Console 106 generates artificial reality content122C, and includes virtual hat 133 at a position that is determinedbased on the location of the head of user 102. Console 106 causes HMD112A to present artificial reality content 122C to user 101 in themanner shown in FIG. 1C.

In some examples, virtual hat 133 may be locked to a position that isdetermined based on the location of HMD 112B, rather than user 102. Insuch an example, user 102 removing HMD 112B and handing HMD 112B toanother user may appear, to user 101 viewing this sequence of eventsthrough HMD 112A, that user 102 is removing virtual hat 133 and handingvirtual hat 133 to the other user.

In still other examples, console 106 may present artificial realitycontent that is locked or presented relative to other physical objectswithin physical environment 120C. In such examples, console 106 maypresent themed versions of physical environment 120C, such that user 101may look at images reflected by mirror 109 and see versions of physicalobjects within physical environment 120C that represent that theme (e.g.western theme, or a theme based on another part of the world, or a“parallel universe” version of physical environment 120C). For instance,user 101 may be in his bedroom, and may look into mirror 109 and see thesame bedroom but with a “western” theme. In response to user inputselecting a theme (e.g., selecting a theme being previewed by looking inthe mirror), the chosen 3D content appears in the 3D physical space ofthe room occupied by user 101, and user 101 can now view directly theselected theme in the physical space, without having to look in mirror109. In some examples, the western theme might change a shoe on thefloor in the user's room to a cowboy boot, or change a baseball hathanging on a hook in the user's room to a cowboy hat. Also, in someexamples, the western theme might add objects to the user's room thatmight be expected to be found in a western-style room or in a roomaugmented for the chosen theme.

In another example that presents a “parallel universe” version ofphysical environment 120C, user 101 may be presented with a userinterface enabling user 101 to “travel” to the parallel universe, whichmay involve applying appropriate virtual clothing content to that user'sappearance (e.g., to the user's reflections in mirror 109) and to theappearance of any other users in the room.

Techniques described herein may enable use of an artificial realitysystem to know who else is sharing the same artificial realityexperience as that user. One way in which such a shared experience maybe indicated is through an artificial reality augmentation (e.g., adistinctive hat) that may be visible to other users in that experience.A hat or other article of clothing chosen or otherwise worn by a usermay have a particular meaning, which may depend on the community orcontext in which the hat is worn. Such content may be customizable bythe user through interactions with a user interface. A hat or otherarticle of clothing may animate or make a sound when the person walks ormoves, or based on the person's posture (e.g., a Santa hat with a bellon the end). Hats or other accessories may be presented based onavailable space or based on attributes of the physical environment(e.g., particular hats available only in high-ceiling rooms). Sucharticles of clothing or other digital enhancements that might be lockedto the user's position (e.g., substantially locked to the user's head)may be used in various artificial reality experiences to signifyattributes of a user, such as where that user was, where he or shecurrently is, what that user knows or has created, or what role such auser is serving or performing.

In other examples, techniques described herein may be used in connectionwith a social event where users are able to choose (e.g., on an opt-inbasis) an artificial reality party hat that is visible only when thatuser is in a designated space for that social event. Such a hat mightdisappear or not be visible when the user is outside that designatedspace. Where multiple users are in front of a mirror, a user interfacemight be presented enabling hats or other content to be changed formultiple users at the same time. In some examples, hats might berandomized or chosen from a set of hats based on the number of userspresent in front of the mirror or based on the time of year or otherevent. In one example that might occur during Christmas season, one userfrom among many may be chosen to wear a Santa hat, and the remainingusers might be outfitted with elf hats. In another example where a musicconcert is on a relevant user's schedule, virtual hats and/oraccessories may be chosen based on the type of music to be played at theconcert or the band performing at concert.

Although techniques described herein are primarily described in terms ofa physical mirror that is affixed to a wall, techniques described hereinmay be applicable in other situations. For example, a personal mirror(e.g., a small hand-held mirror) may be used for similar effects. In theexamples of FIG. 1A, FIG. 1B, and FIG. 1C, for instance, externalsensors 190, cameras 192, and HMDs 112 may track the position of such amirror and the movements of such a mirror. Such a portable mirror mightalso be configured with inertial measurement devices or sensors thatenable tracking. In one example, a portable mirror may be a mobiledevice (e.g., a mobile phone) that allows a user to see him or herselfusing a forward-facing camera.

FIG. 2 is an illustration depicting an example HMD 112 configured tooperate in accordance with the techniques of the disclosure. HMD 112 ofFIG. 2 may be an example of any HMD 112 of FIG. 1A, FIG. 1B, and/or FIG.1C. HMD 112 may be part of an artificial reality system, such asartificial reality systems 100A, 100B, or 100C, or may operate as astand-alone, mobile artificial realty system configured to implement thetechniques described herein. HMD 112 may include a mobile device (e.g.,a smart phone) that is removable from the body of the HMD 112.

In the example of FIG. 2, HMD 112 includes a front rigid body and a bandto secure HMD 112 to a user. In addition, HMD 112 includes aninterior-facing electronic display 203 configured to present artificialreality content to the user. Electronic display 203 may be any suitabledisplay technology, such as liquid crystal displays (LCD), quantum dotdisplay, dot matrix displays, light emitting diode (LED) displays,organic light-emitting diode (OLED) displays, cathode ray tube (CRT)displays, e-ink, or monochrome, color, or any other type of displaycapable of generating visual output. In some examples, the electronicdisplay is a stereoscopic display for providing separate images to eacheye of the user. In some examples, the known orientation and position ofdisplay 203 relative to the front rigid body of HMD 112 is used as aframe of reference, also referred to as a local origin, when trackingthe position and orientation of HMD 112 for rendering artificial realitycontent according to a current viewing perspective of HMD 112 and theuser.

In the example of FIG. 2, HMD 112 further includes one or more sensors206, such as one or more accelerometers (also referred to as inertialmeasurement units or “IMUs”) that output data indicative of currentacceleration of HMD 112, GPS sensors that output data indicative of alocation of HMD 112, radar or sonar sensors that output data indicativeof distances of the HMD 112 from various objects, or other sensors thatprovide indications of a location or orientation of HMD 112 or otherobjects within a physical 3D environment. Moreover, HMD 112 may includeone or more integrated sensor devices 208, such as a microphone, audiosensor, a video camera, laser scanner, Doppler radar scanner, depthscanner, or the like, configured to output audio or image datarepresentative of a surrounding real-world environment. HMD 112 includesan internal control unit 210, which may include an internal power sourceand one or more printed-circuit boards having one or more processors,memory, and hardware to provide an operating environment for executingprogrammable operations to process sensed data and presentartificial-reality content on display 203. Internal control unit 210 maybe part of a removable computing device, such as a smart phone.

Although illustrated in FIG. 2 having a specific configuration andstructure, HMD 112 may take any of a number of forms. For example, insome implementations, HMD 112 might resemble glasses or may have adifferent form. Also, although HMD 112 may be configured with a display203 for presenting representations or images of physical content, inother examples, HMD 112 may include a transparent or partiallytransparent viewing lens, enabling see-through artificial reality (i.e.,“STAR”). Further, HMD may implement features based on wave guides orother STAR technologies.

In accordance with the techniques described herein, control unit 210 isconfigured to present content within the context of a physicalenvironment that includes one or more mirrors. For example, HMD 112 maycompute, based on sensed data generated by motion sensors 206 and/oraudio and image data captured by sensor devices 208, a current pose fora frame of reference of HMD 112. Control unit 210 may include a posetracking unit, which can execute software for processing the sensed dataand/or images to compute the current pose. Control unit 210 may store amaster 3D map for a physical environment and compare processed images tothe master 3D map to compute the current pose. Alternatively, oradditionally, control unit 210 may compute the current pose based onsensor data generated by sensors 206. Based on the computed currentpose, control unit 210 may render artificial reality contentcorresponding to the master 3D map for an artificial realityapplication, and control unit 210 may display the artificial realitycontent via the electronic display 203.

As another example, control unit 210 may generate mapping informationfor the physical 3D environment in which the HMD 112 is operating andsend, to a console or one or more other computing devices (such as oneor more other HMDs), via a wired or wireless communication session(s),the mapping information. In this way, HMD 112 may contribute mappinginformation for collaborate generation of the master 3D map for thephysical 3D environment. Mapping information may include images capturedby sensor devices 208, tracking information in the form of indicationsof the computed local poses, or tracking information that provideindications of a location or orientation of HMD 112 within a physical 3Denvironment (such as sensor data generated by sensors 206), for example.

In some examples, in accordance with the techniques described herein,control unit 210 may peer with one or more controllers for HMD 112(controllers not shown in FIG. 2). Control unit 210 may receive sensordata from the controllers that provides indications of user inputs orcontroller orientations or locations within the physical 3D environmentor relative to HMD 112. Control unit 210 may send representations of thesensor data to a console for processing by the artificial realityapplication, where the indications may be event data for an artificialreality application. Control unit 210 may execute the artificial realityapplication to process the sensor data.

FIG. 3 is a block diagram showing example implementations of an exampleconsole and an example HMD, in accordance with one or more aspects ofthe present disclosure. Although the block diagram illustrated in FIG. 3is described with reference to HMD 112, in other examples, functionsand/or operations attributed to HMD 112 may be performed by a differentdevice or system, such as a user device as referenced in connection withFIG. 1A.

In the example of FIG. 3, HMD 112 includes one or more processors 302and memory 304 that, in some examples, provide a computer platform forexecuting an operation system 305, which may be an embedded and near (orseemingly-near) real-time multitasking operating system. In turn,operating system 305 provides a multitasking operating environment forexecuting one or more software components 307. Processors 302 arecoupled to electronic display 203 (see FIG. 2). HMD 112 is shownincluding motion sensors 206 and sensor devices 208 coupled to processor302, but in other examples, HMD 112 may include neither or merely eitherof motion sensors 206 and sensor devices 208. In some examples,processors 302 and memory 304 may be separate, discrete components. Inother examples, memory 304 may be on-chip memory collocated withprocessors 302 within a single integrated circuit. The memory 304,processors 302, operating system 305, and application engine 340components may collectively represent an example of internal controlunit 210 of FIG. 2.

HMD 112 may include user input devices, such as a touchscreen or otherpresence-sensitive screen example of electronic display 203, microphone,controllers, buttons, keyboard, and so forth. Application engine 340 maygenerate and present a login interface via electronic display 203. Auser of HMD 112 may use the user interface devices to input, using thelogin interface, login information for the user. HMD 112 may send thelogin information to console 106 to log the user into the artificialreality system.

Operating system 305 provides an operating environment for executing oneor more software components, which include application engine 306, whichmay be implemented as any type of appropriate module. Application engine306 may be an artificial reality application having one or moreprocesses. Application engine 306 may send, to console 106 as mappinginformation using an I/O interface (not shown in FIG. 3) via a networkor other communication link, representations of sensor data generated bymotion sensors 206 or images generated by sensor devices 208. Theartificial reality application may be, e.g., a teleconferenceapplication, a gaming application, a navigation application, aneducational application, or training or simulation application, forexample.

Console 106 may be implemented by any suitable computing system capableof interfacing with user devices (e.g., HMDs 112) of an artificialreality system. In some examples, console 106 interfaces with HMDs 112to augment content that may be reflected in a mirror, or to presentartificial reality content triggered by an action or gesture performedin a particular location relative to the mirror. In some examples,console 106 generates, based at least on mapping information receivedfrom one or more HMDs 112, external sensors 190, and/or cameras 192, amaster 3D map of a physical 3D environment in which users, physicaldevices, one or more mirrors, and other physical objects are located. Insome examples, console 106 is a single computing device, such as aworkstation, a desktop computer, a laptop. In some examples, at least aportion of console 106, such as processors 352 and/or memory 354, may bedistributed across one or more computing devices, a cloud computingsystem, a data center, or across a network, such as the Internet,another public or private communications network, for instance,broadband, cellular, Wi-Fi, and/or other types of communicationnetworks, for transmitting data between computing systems, servers, andcomputing devices.

In the example of FIG. 3, console 106 includes one or more processors312 and memory 314 that provide a computer platform for executing anoperating system 316. In turn, operating system 316 provides anoperating environment for executing one or more software components 317.Processors 312 are coupled to I/O interface 315, which provides one ormore I/O interfaces for communicating with external devices, such as akeyboard, game controllers, display devices, image capture devices, andthe like. Moreover, I/O interface 315 may include one or more wired orwireless network interface cards (NICs) for communicating with anetwork, such as network 104 (see, e.g., FIG. 1A). Each of processors302, 312 may comprise any one or more of a multi-core processor, acontroller, a digital signal processor (DSP), an application specificintegrated circuit (ASIC), a field-programmable gate array (FPGA), orequivalent discrete or integrated logic circuitry. Memory 304, 314 maycomprise any form of memory for storing data and executable softwareinstructions, such as random-access memory (RAM), read-only memory(ROM), programmable read-only memory (PROM), erasable programmableread-only memory (EPROM), electronically erasable programmable read-onlymemory (EEPROM), and/or Flash memory. Software components 317 of console106 operate to provide an overall artificial reality application. In theexample of FIG. 3, software components 317 be represented by modules asdescribed herein, including application engine 320, rendering engine322, pose tracker 326, mapping engine 328, and user interface engine329.

Application engine 320 includes functionality to provide and present anartificial reality application, e.g., a teleconference application, agaming application, a navigation application, an educationalapplication, training or simulation applications, and the like.Application engine 320 and application engine 340 may cooperativelyprovide and present the artificial reality application in some examples.Application engine 320 may include, for example, one or more softwarepackages, software libraries, hardware drivers, and/or ApplicationProgram Interfaces (APIs) for implementing an artificial realityapplication on console 106. Responsive to control by application engine320, rendering engine 322 generates 3D artificial reality content fordisplay to the user by application engine 340 of HMD 112.

Rendering engine 322 renders the artificial content constructed byapplication engine 320 for display to user 101 in accordance withcurrent pose information for a frame of reference, typically a viewingperspective of HMD 112, as determined by pose tracker 326. Based on thecurrent viewing perspective, rendering engine 322 constructs the 3D,artificial reality content which may be overlaid, at least in part, uponthe physical 3D environment in which HMD 112 is located. During thisprocess, pose tracker 326 may operate on sensed data received from HMD112, such as movement information and user commands, and, in someexamples, data from external sensors 190 and/or cameras 192 (as shown inFIG. 1A, FIG. 1B, and FIG. 1C) to capture 3D information within thephysical 3D environment, such as motion by HMD 112, a user thereof, acontroller, and/or feature tracking information with respect to the userthereof.

Pose tracker 326 determines information relating to a pose of a userwithin an physical environment. For example, console 106 may receivemapping information from HMD 112, and mapping engine 328 mayprogressively generate a map for an area in which HMD 112 is operatingover time, HMD 112 moves about the area. Pose tracker 326 may localizeHMD 112, using any of the aforementioned methods, to the map for thearea. Pose tracker 326 may also attempt to localize HMD 112 to othermaps generated using mapping information from other user devices. Atsome point, pose tracker 326 may compute the local pose for HMD 112 tobe in an area of the physical 3D environment that is described by a mapgenerated using mapping information received from a different userdevice. Using mapping information received from HMD 112 located andoriented at the computed local pose, mapping engine 328 may join the mapfor the area generated using mapping information for HMD 112 to the mapfor the area generated using mapping information for the different userdevice to close the loop and generate a combined map for the master 3Dmap. Mapping engine 328 stores such information as map data 330. Basedsensed data collected by external sensors 190, cameras 192, HMD 112, orother sources, pose tracker 326 determines a current pose for the frameof reference of HMD 112 and, in accordance with the current pose,provides such information to application engine 320 for generation ofartificial reality content. That artificial reality content may then becommunicated to HMD 112 for display to the user via electronic display203.

Mapping engine 328 may be configured to generate maps of a physical 3Denvironment using mapping information received from user devices.Mapping engine 328 may receive the mapping information in the form ofimages captured by sensor devices 208 at local poses of HMD 112 and/ortracking information for HMD 112, for example. Mapping engine 328processes the images to identify map points for determining topographiesof the scenes in the images and use the map points to generate map datathat is descriptive of an area of the physical 3D environment in whichHMD 112 is operating. Map data 330 may include at least one master 3Dmap of the physical 3D environment that represents a current best map,as determined by mapping engine 328 using the mapping information.

Mapping engine 328 may receive images from multiple different userdevices operating in different areas of a physical 3D environment andgenerate different maps for the different areas. The different maps maybe disjoint in that the maps do not, in some cases, overlap to describeany of the same areas of the physical 3D environment. However, thedifferent maps may nevertheless be different areas of the master 3D mapfor the overall physical 3D environment.

Pose tracker 326 determines information relating to a pose of a userwithin an physical environment. For example, console 106 may receivemapping information from HMD 112, and mapping engine 328 mayprogressively generate a map for an area in which HMD 112 is operatingovertime, HMD 112 moves about the area. Pose tracker 326 may localizeHMD 112, using any of the aforementioned methods, to the map for thearea. Pose tracker 326 may also attempt to localize HMD 112 to othermaps generated using mapping information from other user devices. Atsome point, pose tracker 326 may compute the local pose for HMD 112 tobe in an area of the physical 3D environment that is described by a mapgenerated using mapping information received from a different userdevice. Using mapping information received from HMD 112 located andoriented at the computed local pose, mapping engine 328 may join the mapfor the area generated using mapping information for HMD 112 to the mapfor the area generated using mapping information for the different userdevice to close the loop and generate a combined map for the master 3Dmap. Mapping engine 328 stores that maps as map data 330. Based senseddata collected by external sensors 190, cameras 192, HMD 112, or othersources, pose tracker 326 determines a current pose for the frame ofreference of HMD 112 and, in accordance with the current pose, providessuch information to application engine 320 for generation of artificialreality content. That artificial reality content may then becommunicated to HMD 112 for display to the user via electronic display203.

Mapping engine 328 may use mapping information received from HMD 112 toupdate the master 3D map, which may be included in map data 330. Mappingengine 328 may, in some examples, determine whether the mappinginformation is preferable to previous mapping information used togenerate the master 3D map. For example, mapping engine 328 maydetermine the mapping information is more recent in time, of higherresolution or otherwise better quality, indicates more or differenttypes of objects, has been generated by a user device having higherresolution localization abilities (e.g., better inertial measurementunit or navigation system) or better optics or greater processing power,or is otherwise preferable. If preferable, mapping engine 328 generatesan updated master 3D map from the mapping information received from HMD112. Mapping engine 328 in this way progressively improves the master 3Dmap.

In some examples, mapping engine 328 may generate and store health datain association with different map data of the master 3D map. Forexample, some map data may be stale in that the mapping information usedto generate the map data was received over an amount of time ago, or themap data may be of poor quality in that the images used to the generatethe map data were poor quality (e.g., poor resolution, poor lighting,etc.). These characteristics of the map data may be associated withrelatively poor health. Contrariwise, high quality mapping informationwould be associated with relatively good health. Health values for mapdata may be indicated using a score, a descriptor (e.g., “good”, “ok”,“poor”), a date generated, or other indicator. In some cases, mappingengine 328 may update map data of the master 3D map for an area if thehealth for the map data satisfies a threshold health value (e.g., isbelow a certain score). If the threshold health value is satisfied,mapping engine 328 generates an updated area for the area of the master3D map using the mapping information received from HMD 112 operating inthe area. Otherwise, mapping engine 328 discards the mappinginformation.

In some examples, map data 330 includes different master 3D maps fordifferent areas of a physical 3D environment. Pose tracker 326 maylocalize HMD 112 to a location in one of the areas using images receivedfrom HMD 112. In response, application engine 320 may select the master3D map for the area within which pose tracker 326 localized HMD 112 andsend the master 3D map to HMD 112 for use in the artificial realityapplication. Consequently, HMD 112 may generate and render artificialreality content using the appropriate master 3D map for the area inwhich HMD 112 is located.

In some examples, map data includes different master 3D maps for thesame area of a physical 3D environment, the different master 3D mapsrepresenting different states of the physical environment. For example,a first master 3D map may describe an area at a first time e.g., August2015, while a second master 3D map may describe the area at a secondtime, e.g., October 2016. Application engine 320 may determine to usethe first master 3D map responsive to a request from the user orresponsive to a trigger within an artificial reality application, forinstance. The mapping engine 328 may indicate in map data 330 that thefirst master 3D map is the master 3D map that is to be used forrendering artificial reality content for an artificial realityapplication. In this way, an artificial reality system including console106 can render artificial reality content using historical map datadescribing a physical 3D environment as it appeared in earlier times.This technique may be advantageous for education-related artificialreality applications, for instance.

User interface engine 329 may perform functions relating to generating auser interface when a user is in close proximity to mirror 109 and/orwhen a user performs a gesture or action (e.g., touching the surface ofmirror 109). User interface engine 329 may receive information fromapplication engine 320, pose tracker 326, and/or mapping engine 328 andbased on that information, generate a user interface (e.g., userinterface menu 124 having user interface elements 126). User interfaceengine 329 may output, to rendering engine 322, information about theuser interface so that rendering engine 322 may present the userinterface, overlaid on other physical and/or artificial reality content,at display 203 of HMD 112. Accordingly, user interface engine 329 mayreceive information from and output information to one or more othermodules, and may otherwise interact with and/or operate in conjunctionwith one or more other engines or modules of console 106.

In some examples, such as in the manner described in connection withFIG. 4, some or all of the functionality attributed to pose tracker 326,rendering engine 322, configuration interface 332, classifier 324, andapplication engine 320 may be performed by HMD 112.

Modules or engines illustrated in FIG. 3 (e.g., operating system 316,application engine 320, rendering engine 322, pose tracker 326, mappingengine 328, user interface engine 329, operating system 305, andapplication engine 306), FIG. 4, and/or illustrated or describedelsewhere in this disclosure may perform operations described usingsoftware, hardware, firmware, or a mixture of hardware, software, andfirmware residing in and/or executing at one or more computing devices.For example, a computing device may execute one or more of such moduleswith multiple processors or multiple devices. A computing device mayexecute one or more of such modules as a virtual machine executing onunderlying hardware. One or more of such modules may execute as one ormore services of an operating system or computing platform. One or moreof such modules may execute as one or more executable programs at anapplication layer of a computing platform. In other examples,functionality provided by a module could be implemented by a dedicatedhardware device.

Although certain modules, data stores, components, programs,executables, data items, functional units, and/or other items includedwithin one or more storage devices may be illustrated separately, one ormore of such items could be combined and operate as a single module,component, program, executable, data item, or functional unit. Forexample, one or more modules or data stores may be combined or partiallycombined so that they operate or provide functionality as a singlemodule. Further, one or more modules may interact with and/or operate inconjunction with one another so that, for example, one module acts as aservice or an extension of another module. Also, each module, datastore, component, program, executable, data item, functional unit, orother item illustrated within a storage device may include multiplecomponents, sub-components, modules, sub-modules, data stores, and/orother components or modules or data stores not illustrated.

Further, each module, data store, component, program, executable, dataitem, functional unit, or other item illustrated within a storage devicemay be implemented in various ways. For example, each module, datastore, component, program, executable, data item, functional unit, orother item illustrated within a storage device may be implemented as adownloadable or pre-installed application or “app.” In other examples,each module, data store, component, program, executable, data item,functional unit, or other item illustrated within a storage device maybe implemented as part of an operating system executed on a computingdevice.

FIG. 4 is a block diagram depicting an example of a user device for anartificial reality system, in accordance with one or more aspects of thepresent disclosure. In FIG. 4, HMD 112 may operate as a stand-alonedevice, i.e., not tethered to a console, and may represent an instanceof any of the user devices, including HMDs 112 described in connectionwith FIG. 1A, FIG. 1B, and FIG. 1C. Although device 112 illustrated inFIG. 4 is primarily described as a head-mounted device, the deviceillustrated in FIG. 4 may, in other examples, be implemented as adifferent device, such as tablet computer, for instance. In the specificexample of FIG. 4, however, and in a manner similar to FIG. 3, HMD 112includes one or more processors 302 and memory 304 that, in someexamples, provide a computer platform for executing an operation system305, which may be an embedded multitasking operating system. In turn,operating system 305 provides an operating environment for executing oneor more software components 417. Moreover, processor(s) 302 are coupledto electronic display 203, motion sensors 206, and sensor devices 208.

In the example of FIG. 4, software components 417 operate to provide anoverall artificial reality application. In this example, softwarecomponents 417 include application engine 420, rendering engine 422,pose tracker 426, mapping engine 428, and user interface (UI) engine429. In various examples, software components 417 operate similar to thecounterpart components of console 106 of FIG. 3 (e.g., applicationengine 320, rendering engine 322, pose tracker 326, mapping engine 328,and user interface engine 329).

One or more aspects of FIG. 4 may be described herein within the contextof other Figures, including FIG. 1A, FIG. 1B, and FIG. 1C. In variousexamples, HMD 112 may generate map information, determine a pose, detectinput, identify objects that are reflected in mirror 109, and presentartificial reality content.

In accordance with one or more aspects of the present disclosure, HMD112 of FIG. 1A and FIG. 4 may generate map information. For instance,with reference to FIG. 1A and FIG. 4, each of external sensors 190,cameras 192, sensor devices 208 collect information about physicalenvironment 120A. External sensors 190 and cameras 192 communicates theinformation each collects to HMD 112, and such information may becommunicated to HMD 112 over network 104 or through other means. HMD 112receives information from external sensors 190 and/or cameras 192 andoutputs to mapping engine 428 information about physical environment120A. Sensor devices 208 of HMD 112 also collects information aboutphysical environment 120A, and outputs to mapping engine 428 informationabout physical environment 120A. Mapping engine 428 determines, based onthe information received from external sensors 190, cameras 192, and/orsensor devices 208, a map of physical environment 120A. Mapping engine428 stores information about the map as map data 430.

HMD 112 may determine pose information. For instance, referring again toFIG. 1A and FIG. 4, motion sensor 206 and/or sensor devices 208 detectinformation about the position, orientation, and/or location of HMD 112.Pose tracker 426 receives from mapping engine 428 information about theposition, orientation, and/or location of HMD 112. Pose tracker 426determines, based on this information a current pose for a frame ofreference of HMD 112.

HMD 112 may determine which objects are reflected in mirror 109. Forinstance, again with reference to FIG. 1A and FIG. 4, mapping engine 428analyzes map data 430. Mapping engine 428 determines, based on therelative positions of the physical objects and mirror 109 within 120A,which physical objects within physical environment 120A are reflected inmirror 109. Mapping engine 428 may make such a determination from theperspective of one or more locations, but in the example of FIG. 1A,mapping engine 428 at least determines which physical objects withinphysical environment 120A are reflected within mirror 109 for theperspective of HMD 112.

HMD 112 may generate artificial reality content 122A. For instance,referring again to FIG. 1A and FIG. 4, mapping engine 428 outputs, toapplication engine 420, information about mapping information forphysical environment 120A. Pose tracker 426 outputs, to applicationengine 420, information about the current pose determined for a frame ofreference of HMD 112. Application engine 420 determines, based on themapping and pose information, that from the perspective of user 101, HMD112 is visible in mirror 109 (i.e., user 101 can see HMD 112 reflectedby mirror 109). Application engine 420 determines, based on aconfiguration, type of application, execution context, user input,attributes of user 101, or other information, that it is appropriate togenerate artificial reality content that includes a hat (i.e., virtualhat 123) located on the head of user 101. Application engine 420determines an appropriate form for virtual hat 123, again based on aconfiguration, the type of application being executed, executioncontext, user input, attributes of user 101, or other information.Application engine 420 determines the orientation and/or position ofvirtual hat 123 within artificial reality content 122A based on the poseinformation associated with user 101. Application engine 420 generates,again based on the mapping and pose information, artificial realitycontent 122A. When generating artificial reality content 122A,application engine 420 overlays virtual hat 123 on the head of user 101.

HMD 112 may render artificial reality content 122A. For instance,referring to FIG. 1A and FIG. 4, application engine 420 outputsinformation about artificial reality content 122A to rendering engine422, where such information includes information about the virtual hat123 overlay. Rendering engine 422 causes artificial reality content 122Ato presented at display 203 in the manner shown in FIG. 1A.

HMD 112 may cause a user interface to be presented. For instance,referring now to FIG. 1B and FIG. 4. Mapping engine 428 of HMD 112receives information from external sensors 190 and/or cameras 192 aboutmovements by user 101. Alternatively, or in addition, mapping engine 428receives information from motion sensors 206 and/or sensor devices 208about movements by user 101. Mapping engine 428 determines, based on thereceived information, that user 101 has touched mirror 109 at touchpoint 119. Mapping engine 428 outputs information about the movement touser interface engine 429. User interface engine 429 generates, based onthe information received from mapping engine 428 and other information(e.g., application context or state), information sufficient to generateuser interface menu 124. User interface engine 429 outputs informationabout user interface menu 124 to application engine 420. Applicationengine 420 updates artificial reality content 122A to include userinterface menu 124, thereby generating artificial reality content 122B.

HMD 112 may render artificial reality content 122B. For instance, againreferring to FIG. 1B and FIG. 4, application engine 420 outputsinformation about artificial reality content 122B to rendering engine422, where such information includes information about user interfacemenu 124. Rendering engine 422 causes artificial reality content 122B topresented at display 203 in the manner shown in FIG. 1B.

HMD 112 may perform further operations in response to interactions withuser interface menu 124. For instance, still referring to FIG. 1B andFIG. 4, HMD 112 may detect movements by user 101 that it determinescorresponds to selection of one or more user interface elements 126within user interface menu 124. Application engine 420 may, in responseto such movements, perform one or more operations. In some examples,such operations may cause user interface engine 429 to generate furtheruser interfaces. In such examples, application engine 420 updatesartificial reality content 122B, and causes rendering engine 422 topresent the updated content to the user at display 203.

FIG. 5A and FIG. 5B are example diagrams illustrating one possibletechnique for identifying physical objects that are reflected in amirror or other reflective surface, in accordance with one or moreaspects of the present disclosure. FIG. 5A illustrates diagram 500including mirror 509 mounted on a wall coincident with plane 519, withthe reflective surface of mirror 509 facing in the direction indicatedby the arrow. In FIG. 5A, the mirror and/or wall defines plane 519.Since the perspective of FIG. 5A and FIG. 5B are from above the wall,the mirror 509 and plane 519 are shown as a line. Observer 510 is shownin diagram 500, and the position of observer 510 may correspond to thatof a camera, user, or other object. For instance, in FIG. 1A, observer510 may be HMD 112 (or user 101).

Physical objects 516A and 516B (collectively “physical objects 516”) arealso shown in FIG. 5A, and for ease of illustration, each is illustratedas simply circle adjacent to a smaller circle, indicating a particularorientation relative to plane 519. For physical object 516A, areflection of physical object 516A across plane 519 is illustrated asphysical object 516A′. Similarly, for physical object 516B, a reflectionof physical object 516B across plane 519 is illustrated as physicalobject 516B′. Each such reflection is indicated by one of trace lines512, and each of trace lines 512 is perpendicular to plane 519.

FIG. 5B is similar to FIG. 5A, but also includes sight lines 514extending from observer 510 to each of the reflections of physicalobjects 516 (physical objects 516′). Sight lines 514 are used in FIG. 5Bto determine whether a reflection of any of physical objects 516 isvisible to observer 510. Determining whether a reflection of each ofphysical objects 516 is visible to observer 510 involves evaluatingwhether each sight line 514 intersects plane 519 the two-dimensionalregion defined by mirror 509.

To determine which of physical objects 516 have reflections visible toobserver 510, HMD 112 of FIG. 4 may generate a map of a physical areacorresponding to diagram 500. For instance, in an example that can bedescribed with reference to FIG. 5A and FIG. 4, mapping engine 428generates map data 430 based on information received from sensor devices208 or other sources. Application engine 420 identifies, based on mapdata 430, each of physical objects 516 and the positions of each ofphysical objects 516 within the physical environment associated withdiagram 500. Application engine 420 reflects each of physical objects516 across plane 519. To perform the reflection for each of physicalobjects 516, application engine 420 may cause mapping engine 428 toperform calculations and update map data 430 to reflect the positions ofreflected physical objects 516′ that are shown in FIG. 5A. Applicationengine 420 determines, based on such calculations, the position andscale of each of physical objects 516. Application engine 420 applies anegative scale to each of physical objects 516 along the normal to plane519, thereby determining the orientation of each of physical objects516. In some examples, scale changes can be simulated by moving virtualcontent as user moves, which then appears as a scale change within the3D rendering environment. Negative scale can be used on the reflectedvirtual object though to change the orientation of the object. Anothermethod for handling scale changes involves having two meshes, one is areflected mesh, one is not.

HMD 112 may determine which of physical objects 516 are visible inmirror 509. For instance, referring now to FIG. 5B and FIG. 4,application engine 420 determines sight lines 514 from observer 510 toeach of reflected physical objects 516′. Application engine 420determines which of sight lines 514 intersect the region defined by thetwo-dimensional boundaries of mirror 509. For any reflections ofphysical objects (516′) having a sight line 514 intersecting the regiondefined by the two-dimensional boundaries of mirror 509, applicationengine 420 identifies the corresponding physical objects 516 as having areflection visible to observer 510. For any reflections of physicalobjects (516′) having a sight line 514 not intersecting the regiondefined by the two-dimensional boundary of mirror 509, applicationengine 420 identifies the corresponding physical objects 516 as nothaving a reflection visible to observer 510. In the example of FIG. 5B,application engine 420 determines that from the perspective of observer510, a reflection of physical object 516A is visible within mirror 509,since sight line 514 associated with physical object 516A (to 516A′)intersects mirror 509. Application engine 420 determines that areflection of physical object 516B, however, is not visible in mirror509.

In some examples, application engine 420 may determine whether areflection of each of physical objects 516 is visible in mirror 509 on acenter-to-center basis (i.e., center of observer 510 to center ofphysical object 516), on a pixel-by-pixel basis, using a sphere or boxtrace, or using other methods. In some cases, application engine 420 maydetermine that some portion of a particular physical object 516 isvisible in mirror 509, but not all of physical object 516 is visible. Insuch an example, and where artificial reality content is presented inconnection with (e.g., at a locked position with) reflected images ofobjects, application engine 420 may generate only a portion of theartificial reality content for objects not fully reflected in mirror509. Some physical objects 516 may have reflections that are “clipped”by the boundaries of mirror 509, and artificial reality content that ispresented with reference to the position of such physical objects 516might also be correspondingly “clipped” as appropriate.

FIG. 6 is an example diagram illustrating another possible technique foridentifying physical objects that are reflected in a mirror or otherreflective surface, in accordance with one or more aspects of thepresent disclosure. FIG. 6 is similar to FIG. 5A and FIG. 5B in that itincludes mirror 509 mounted on a wall coincident with plane 519, withthe reflective surface of mirror 509 facing in the direction indicatedby the arrow. The surface of mirror 509 and/or the wall that mirror 509is mounted on defines plane 519. As in FIG. 5A and FIG. 5B, theperspective of FIG. 6 is from directly above the wall, so the mirror 509and plane 519 are shown as a line. Observer 610 is also shown in diagram600, and like observer 510 of FIG. 5A and FIG. 5B, may represent to theperspective of a camera, user, or other object.

In the example of FIG. 6, to determine which of physical objects 516have reflections visible to observer 510, HMD 112 may generate a map ofa physical area corresponding to diagram 600. For instance, withreference to FIG. 6 and FIG. 4, mapping engine 428 generates map data430. Application engine 420 identifies, based on map data 430, physicalobjects 516A and 516B and the positions of each of physical objects 516within the physical environment associated with diagram 600. Applicationengine 420 reflects observer 610 across plane 519 as observer 610′,indicated by trace 612 in FIG. 6. To reflect observer 610 across plane519, application engine 420 may cause mapping engine 428 to performcalculations and update map data 430 to reflect the position of observer610′ as illustrated in FIG. 6.

HMD 112 may determine which of physical objects 516 are visible inmirror 509. For instance, referring again to FIG. 6 and FIG. 4,application engine 420 determines sight lines 614 from reflectedobserver 610′ to the outer boundaries of mirror 509. Application engine420 constructs a three-dimensional frustrum 620 from thethree-dimensional space defined by the point at which observer 610′ ispositioned and the two-dimensional region defined by the boundaries ofmirror 509. Where mirror 509 is a square, for example, frustrum 620 hasa form similar to a clipped pyramid (often an oblique pyramid) thatbegins at mirror 509 and extends into the room in which mirror 509 islocated, with one base of the frustrum being coincident with the surfaceof mirror 509. Where mirror 509 is circular, on the other hand, frustrum620 is conical (often an oblique conical frustrum) with one base offrustrum 620 again being coincident with the surface of mirror 509.

Application engine 420 determines whether a given physical object 516has a reflection visible to observer 610 by determining whether thatphysical object 516 is within frustrum 620. In the example of FIG. 6,application engine 420 determines that since physical object 516A iswithin frustrum 620, so engine 420 determines that a reflection ofphysical object 516A is visible within mirror 509 from the perspectiveof observer 610. Application engine 420 further determines that sincephysical object 516B is not within frustrum 620, no visible reflectionof physical object 516B is shown within mirror 509 from the perspectiveof observer 610.

Although FIG. 5A, 5B and FIG. 6 are described in terms of determiningwhether a given object is reflected in a mirror based on calculatingpositions of objects in a map, it may be possible to make suchdeterminations in another way. For instance, application engine 420 may,alternatively or in addition, analyze an image of a mirror and based onthe appearance of any reflected objects in the mirror, determine whatphysical objects correspond to those reflections.

FIG. 7A, FIG. 7B, and FIG. 7C are conceptual diagrams illustratingexample techniques for enabling multiple users to communicate and/orperform other operations using a mirror, in accordance with one or moreaspects of the present disclosure. Each of FIG. 7A, FIG. 7B, and FIG. 7Cillustrates physical environment 720A and physical environment 720B.Physical environment 720A is occupied by user 101 and physicalenvironment 720B is occupied by user 102. Physical environment 720Aincludes mirror 109A and physical environment 720 includes mirror 109B.Other physical objects are present in each of physical environments 720Aand 720B, including lamp 107, window 108, object 141, and object 142.

In physical environment 720A, user 101 can see his reflection in mirror109A, and user 101 wears HMD 112A. HMD 112A is configured in a mannersimilar to that described in connection with FIG. 4. In physicalenvironment 720B, user 102 can see his reflection in mirror 109B, anduser 102 wears 112B. Like HMD 112A, HMD 112B is configured in a mannersimilar to that described in connection with FIG. 4.

In each of FIG. 7A, FIG. 7B, and FIG. 7C, user 101 wears a shirt bearingthe “1” numeral and having one stripe on the right sleeve. In each ofFIG. 7A, FIG. 7B, and FIG. 7C, user 102 wears a shirt bearing the “2”numeral and having two stripes on the right sleeve.

In the example of FIG. 7A, and in accordance with one or more aspects ofthe present disclosure, HMD 112A may determine that user 101 seeks tocommunicate with user 102. For instance, in an example that can bedescribed with reference to FIG. 7A and FIG. 4, motion sensors 206 ofHMD 112A detect motion and sensor devices 208 capture images. Motionsensors 206 and sensor devices 208 output information about the detectedmotion and captured images to pose tracker 426. Pose tracker 426determines, based on the information, that user 101 has performed agesture touching mirror 109A. Pose tracker 426 outputs information aboutthe gesture to application engine 420. Application engine 420 determinesthat the gesture corresponds to a request to initiate communication withuser 102. In some examples, the gesture may involve interactions with auser interface, such as that illustrated in FIG. 1B.

HMD 112A may establish communication with HMD 112B. For instance, withreference to FIG. 7A and FIG. 4, application engine 420 determines thatuser 102 is wearing HMD 112B. Application engine 420 causes HMD 112A tocommunicate, over network 104, with HMD 112B. Application engine 420causes HMD 112A to exchange information with HMD 112B over network 104,and thereby establishes communication between HMD 112A and HMD 112B.

HMD 112A may present artificial reality content 722A to user 101. Forinstance, still referring to FIG. 7A and FIG. 4, HMD 112B capturesimages of user 102 that are reflected by mirror 709B. HMD 112Bcommunicates the images over network 104 to HMD 112A. Application engine420 of HMD 112A uses the images to construct artificial reality content722A, showing an image of user 102 overlaid on the reflected image thatuser 101 would otherwise see in mirror 709A. Application engine 420outputs information about artificial reality content 722A to renderingengine 422. Rendering engine 422 causes artificial reality content 722Ato be presented at display 203 (within HMD 112A) in the manner shown inFIG. 7A.

HMD 112A may enable user 101 and user 102 to engage in an audio and/orvideo conference. For instance, still referring to FIG. 7A, sensordevices 208 capture audio information and output the audio informationto application engine 420. Sensor devices 208 also capture images ofuser 101 reflected in mirror 709A, and output image data to applicationengine 420. Application engine 420 receives the audio and image data andcauses HMD 112A to communicate the information over network 104 to HMD112B. HMD 112B performs a similar task, capturing audio and videoinformation of user 102 within physical environment 720B. HMD 112Areceives, over network 104, the audio and video information from HMD112B. Application engine 420 uses the information to update artificialreality content 722A. Application engine 420 outputs information torendering engine 422. Rendering engine 422 causes artificial realitycontent 722A to be presented at display 203 (within HMD 112A), providinguser 101 with a substantially continuous view of user 102, as reflectedby mirror 709B at physical environment 720B. If both HMD 112A and HMD112B similarly present artificial reality content in the mannerdescribed, HMD 112A and HMD 112B thereby enable user 101 and user 102 toengage in an audio and/or video conference where users 101 and 102 cansee and hear each other in mirrors 709A and 709B.

In some examples, HMD 112A and HMD 112B may exchange images that includeuser 101 and user 102 only when each of user 101 and user 102 arepresent in front of their respective mirrors 709. In other words, eventhough HMD 112A and HMD 112B may have access to images within physicalenvironments 720A and 720B of user 101 and user 102, respectively, evenwhen user 101 and user 102 are not in front of a mirror, each of HMD112A and HMD 112B might, at times, not display such images during avideo conference.

Further, although described in terms of two users engaging in a videoconference, techniques described herein may apply to multiple usersengaged in a video conference. For instance, a group of users may bepresent in front of the same mirror or multiple users may be present atseparate mirrors in different physical environments.

FIG. 7B is a conceptual diagram illustrating artificial reality content722B, which is an alternative version of artificial reality content 722Aas presented by HMD 112A in FIG. 7A. In the example described above inconnection with FIG. 7A, HMD 112B captures reflected images of user 102as seen by HMD 112B in mirror 709B. HMD 112B communicates those imagesover network 104 to HMD 112A, and HMD 112A uses the images to presentartificial reality content 722A, as shown in FIG. 7A. In the example ofFIG. 7B, HMD 112A adjusts the images received from HMD 112B so that theyare not presented as reflected or mirror-image versions of user 102.Instead, in the example of FIG. 7B, HMD 112A reverses the imagesreceived from HMD 112B so that they are presented as artificial realitycontent 722B without a mirror-image effect. Note, for example, that thenumeral “2” on the shirt worn by user 102 is presented in a non-reversedway, as is the numeral “2” printed on object 142 present within physicalenvironment 720B. And although HMD 112A is described as performingprocessing to adjust the images captured in physical environment 720B toreverse the “mirror-image” effect, such processing may be done by adifferent device or system, including by HMD 112B.

Further, HMD 112A may further process images captured in physicalenvironment 720B to generate artificial reality content 722B that has adifferent version of user 102 overlaid on the images of user 102. Forinstance, while in some examples, HMD 112A may generate photorealisticversion of user 102, in other examples, HMD 112A may generate an avatar,skeleton, just the virtual hat user 102 is wearing, or any otherappropriate representation of user 102. HMD 112A may cause renderingengine 422 to present such content at display 203 within 112A.

FIG. 7C is a conceptual diagram illustrating artificial reality content722C presented by HMD 112B, which shows images of user 101 captured byHMD 112A at physical environment 720A. In FIG. 7C, HMD 112B receives,over network 104, images of user 101 captured by HMD 112A at physicalenvironment 720A. HMD 112B uses the images to generate artificialreality content 722B that includes images of user 101 captured atphysical environment 720A. HMD 112B may adjust the images to reverse any“mirror-image” effect that may be present in the images received fromHMD 112A. HMD 112B may cause artificial reality content 722B to bepresented to user 102 in HMD 112B. HMD 112B may generate substantiallycontinuous updates to artificial reality content 722B, and present suchartificial reality content 722B to user 102, as illustrated in FIG. 7C.

In some examples, HMD 112A may determine that user 101 seeks to transferdata from 112A to user 102 (wearing HMD 112B). For instance, withreference to FIG. 7C and FIG. 4, motion sensors 206 of HMD 112A detectmotion and sensor devices 208 capture images. Motion sensor 206 andsensor devices 208 output information about the detected motion andcaptured images to pose tracker 426. Pose tracker 426 determines, basedon the information, that user 101 has performed a gesture (or a seriesof gestures) that involves touching mirror 109A at touch point 719A.Pose tracker 426 outputs information about the gesture to applicationengine 420. Application engine 420 determines that the gesturecorresponds to a request to transfer data (e.g., a file or other media)from HMD 112A to HMD 112B.

HMD 112A may, responsive to the gesture, perform the transfer from HMD112A to HMD 112B without requiring input from user 102. For instance,with reference to FIG. 7C and FIG. 4, and in such an example, 112Atransfers the data identified by the gesture to HMD 112B over network104. HMD 112B receives the data and stores the data. HMD 112B may updateartificial reality content 722C to include content or a notificationindicating that data was received. HMD 112B presents the updatedartificial reality content 722C to the user 102 within HMD 112B.

In a different example, HMD 112A may require confirmation input fromuser 102 before performing the transfer to HMD 112B. For instance, againwith reference to FIG. 7C and FIG. 4, and in response to the gesturerequesting the data transfer, application engine 420 causes HMD 112A tooutput a signal over network 104. HMD 112B detects the signal anddetermines that the signal corresponds to a request to receive data fromHMD 112A. HMD 112B presents a user interface, within artificial realitycontent 722C, prompting user 102 to accept the request. HMD 112B furtherprompts user 102 to accept the request by touching touch point 719B onmirror 709B, which may, in some examples, be a point on mirror 709B thatpositionally corresponds to touch point 719A on mirror 709A previouslytouched by user 101. Specifically, in the example of FIG. 7C, HMD 112Bprompts user 102 to touch the appropriate location on mirror 709B byincluding user interface element 726 within artificial reality content722C, where user interface element 726 is presented within artificialreality content 722C at the appropriate place at which user 102 shouldtouch mirror 709B to indicate acceptance. HMD 112B subsequently detectsinput from user 102. In some examples, HMD 112B determines that theinput includes user 102 touching touch point 719B to thereby indicateacceptance of the file transfer. In response to such input, HMD 112Bcommunicates with HMD 112A to signal acceptance of the file transfer,thereby causing HMD 112A to transfer the file to HMD 112B. In otherexamples, HMD 112B might determine that user 102 has not indicatedacceptance of the file transfer, and in such an example, HMD 112A mightnot transfer the data.

In the described example, user 101 transfers data from HMD 112A to HMD112B through a touch interface involving mirror 709A and mirror 709B. Insome implementations, such a process may be perceived to be a useful andintuitive way for user 101 and user 102 to transfer data. User interfaceelement 726 may be illustrated within HMD 112A and HMD 112B as a cube orother three-dimensional object that is handed between users 101 and 102,providing further parallels to real-world interactions. Accordingly,users 101 and 102 may perceive such interactions with mirror 709A andmirror 709B to be a natural extension of real-world actions (e.g.,handing an physical object to another person), particularly where touchpoint 719A and touch point 719B correspond to similar points on mirror709A and mirror 709B, respectively.

FIG. 8 is a conceptual diagram illustrating an example technique forproviding movement instruction using a mirror, in accordance with one ormore aspects of the present disclosure. FIG. 8 illustrates physicalenvironment 820 including mirror 809. Physical environment 820 isillustrated as a room, occupied by user 101 wearing HMD 112, that issimilar to other physical environments illustrated herein, and mayinclude physical objects similar to others described herein. In theexample illustrated in FIG. 8, HMD 112 may present artificial realitycontent 822 that provides movement instruction based on movementsperformed by user 101 as reflected in mirror 809. Some exercise, dance,and other movements are often performed in front of a mirror, andpresenting artificial reality content that provides useful informationto user 101 while such movements are being performed may be perceived bya user as a natural or intuitive extension of such practices. Othermovements not typically performed in front of a mirror may neverthelessbe presented in a similar manner, such as providing out-of-box productinstructions, quick start instructions, or how-to instructions.

In the example of FIG. 8, and in accordance with one or more aspects ofthe present disclosure, HMD 112 may monitor movements performed by user101. For instance, with reference to FIG. 8 and FIG. 4, motion sensors206 of HMD 112A detect motion and sensor devices 208 capture images.Motion sensors 206 and sensor devices 208 output information about thedetected motion and captured images to pose tracker 426. Pose tracker426 determines, based on the information, that user 101 is performing aseries of movements. Pose tracker 426 outputs information about theseries of movements to application engine 420.

HMD 112 may compare detected movements to a model set of movements. Forinstance, still referring to FIG. 8 and FIG. 4, application engine 420analyzes the movements and determines a model set of movements. In someexamples, application engine 420 may determine the model set ofmovements based on an analysis of the detected series of movements. Inother examples, application engine 420 may determine the model set ofmovements based on input (e.g., previously detected input where user 101indicates a particular dance routine that he is performing). Applicationengine 420 compares the detected series of movements to the model set ofmovements.

HMD 112 may generate content that assists user 101 in performingmovements. For instance, again referring to FIG. 8 and FIG. 4,application engine 420 identifies differences between the detectedseries of movements and the model set of movements. Application engine420 generates artificial reality content 822 providing feedback aboutwhether the detected movements match or sufficiently match the model setof movements. In some examples, the detected movements might notsufficiently match the model set of movements. In such an example,application engine 420 may generate artificial reality content 822indicating one or more differences between the compared movements. Asillustrated in FIG. 8, artificial reality content 822 may includemovement instruction 826 as to how user 101 might modify his movementsto be closer to the model set of movements. Alternatively, or inaddition, artificial reality content 822 may include content (e.g.,adjusted arm position 824) indicating a particular motion or movementthat may improve the degree to which movements performed by user 101match those of the model set of movements.

In some examples, artificial reality content 822 may present artificialreality content overlaid on images of user 101 that may include a dancecostume, or workout clothes. In other examples, such content may includeor be modeled after user 101 as an avatar, a skeleton, or other digitalrepresentation of user 101; in still other examples, such content mightbe modeled after a celebrity athlete or other known person.

FIG. 9 is a flow diagram illustrating operations performed by an exampleconsole 106 in accordance with one or more aspects of the presentdisclosure. FIG. 9 is described below within the context of artificialreality system 100A of FIG. 1. In other examples, operations describedin FIG. 9 may be performed by one or more other components, modules,systems, or devices. Further, in other examples, operations described inconnection with FIG. 9 may be merged, performed in a differencesequence, omitted, or may encompass additional operations notspecifically illustrated or described.

In the process illustrated in FIG. 9, and in accordance with one or moreaspects of the present disclosure, console 106 may determine a map of aphysical environment (901). For example, with reference to FIG. 1A, eachof HMD 112, external sensors 190, and/or cameras 192 capture imageswithin physical environment 120A. Console 106 receives such images anddetermines the position of physical objects within physical environment120A, including user 101, HMD 112, and mirror 109. Console 106 generatesmap data (e.g., map data 330 in FIG. 3) describing the physicalenvironment.

Console 106 may identify any physical objects reflected in the mirror(902). For example, in FIG. 1A, console 106 determines which physicalobjects are, from the perspective of HMD 112, visible in mirror 109. Insome examples, determining which physical objects are visible in mirror109 may involve one or more of the techniques described in connectionwith FIG. 5A, FIG. 5B, and/or FIG. 6. In the example of FIG. 1A, console106 determines that images of user 101 and object 141 are visible inmirror 109.

Console 106 may generate artificial reality content for the reflectedobjects (903). For example, in FIG. 1A, console 106 generates artificialreality content 122A by overlaying virtual content on images reflectedin mirror 109. In the example of FIG. 1A, console 106 overlays virtualhat 123 on the image of the user's head. In some examples, virtual hat123 is locked to that user's head (or to HMD 112) so that when user 101(or HMD 112 moves), console 106 updates artificial reality content 122Ato keep virtual hat 123 positioned relative to the user's head, in amanner consistent with how a physical hat would move in such asituation.

Console 106 may present artificial reality content overlaid on thereflections in the mirror (904). For example, in FIG. 1A, console 106causes HMD 112 to present artificial reality content 122A to user 101within HMD 112 in the manner shown in FIG. 1A.

The techniques described in this disclosure may be implemented, at leastin part, in hardware, software, firmware or any combination thereof. Forexample, various aspects of the described techniques may be implementedwithin one or more processors, including one or more microprocessors,DSPs, application specific integrated circuits (ASICs), fieldprogrammable gate arrays (FPGAs), or any other equivalent integrated ordiscrete logic circuitry, as well as any combinations of suchcomponents. The term “processor” or “processing circuitry” may generallyrefer to any of the foregoing logic circuitry, alone or in combinationwith other logic circuitry, or any other equivalent circuitry. A controlunit comprising hardware may also perform one or more of the techniquesof this disclosure.

Such hardware, software, and firmware may be implemented within the samedevice or within separate devices to support the various operations andfunctions described in this disclosure. In addition, any of thedescribed units, modules or components may be implemented together orseparately as discrete but interoperable logic devices. Depiction ofdifferent features as modules or units is intended to highlightdifferent functional aspects and does not necessarily imply that suchmodules or units must be realized by separate hardware or softwarecomponents. Rather, functionality associated with one or more modules orunits may be performed by separate hardware or software components orintegrated within common or separate hardware or software components.

The techniques described in this disclosure may also be embodied orencoded in a computer-readable medium, such as a computer-readablestorage medium, containing instructions. Instructions embedded orencoded in a computer-readable storage medium may cause a programmableprocessor, or other processor, to perform the method, e.g., when theinstructions are executed. Computer readable storage media may includerandom access memory (RAM), read only memory (ROM), programmable readonly memory (PROM), erasable programmable read only memory (EPROM),electronically erasable programmable read only memory (EEPROM), flashmemory, a hard disk, a CD-ROM, a floppy disk, a cassette, magneticmedia, optical media, or other computer readable media.

As described by way of various examples herein, the techniques of thedisclosure may include or be implemented in conjunction with anartificial reality system. As described, artificial reality is a form ofreality that has been adjusted in some manner before presentation to auser, which may include, e.g., a virtual reality (VR), an augmentedreality (AR), a mixed reality (MR), a hybrid reality, or somecombination and/or derivatives thereof. Artificial reality content mayinclude completely generated content or generated content combined withcaptured content (e.g., real-world photographs). The artificial realitycontent may include video, audio, haptic feedback, or some combinationthereof, and any of which may be presented in a single channel or inmultiple channels (such as stereo video that produces athree-dimensional effect to the viewer). Additionally, in some examples,artificial reality may be associated with applications, products,accessories, services, or some combination thereof, that are, e.g., usedto create content in an artificial reality and/or used in (e.g., performactivities in) an artificial reality. The artificial reality system thatprovides the artificial reality content may be implemented on variousplatforms, including a head-mounted display (HMD) connected to a hostcomputer system, a standalone HMD, a mobile device or computing system,or any other hardware platform capable of providing artificial realitycontent to one or more viewers.

What is claimed is:
 1. A system comprising: an image capture systemconfigured to capture image data representative of a physicalenvironment that includes a reflective surface and a plurality ofobjects, wherein the image capture system includes a head-mounteddisplay (HMD) worn by a first user; a mapping engine configured todetermine, based on the image data, a map of the physical environmentincluding position information about the HMD, the reflective surface,and each of the plurality of physical objects; and an application engineconfigured to: enable communication over a network between the firstuser and a second user that is in a second physical environment,determine, based on the position of the HMD and the reflective surface,whether the first user is located in front of the reflective surface,responsive to determining that the first user is located in front of thereflective surface, output, over the network, a plurality of images ofthe first user, and responsive to determining that the first user is notlocated in front of the reflective surface, refrain from outputtingimages of the first user.
 2. The system of claim 1, wherein the physicalenvironment is a first physical environment, wherein the HMD worn by thefirst user is a first HMD, wherein the reflective surface is a firstreflective surface, and wherein the application engine is furtherconfigured to: receive image data captured by a second HMD worn by thesecond user in the second physical environment, wherein the image datacaptured by the second HMD includes images reflected by a secondreflective surface in the second physical environment, and wherein thesecond physical environment is located a remote distance from the firstphysical environment.
 3. The system of claim 2, wherein the applicationengine is further configured to: generate, based on the received imagedata, artificial reality content that includes an image representing thesecond user.
 4. The system of claim 3, wherein to output the pluralityof images of the first user, the application engine is furtherconfigured to: enable the second HMD worn by the second user in thesecond physical environment to present, to the second user, image datarepresentative of the first physical environment.
 5. The system of claim1, further comprising: a rendering engine configured to render, fordisplay at the HMD, the artificial reality content overlaid on an imageof the reflective surface.
 6. The system of claim 1, wherein to outputthe plurality of images of the first user, the application engine isfurther configured to: output images that include image datarepresentative of the first physical environment.
 7. The system of claim6, wherein the image capture system includes: a plurality of cameraspositioned within the physical environment configured to capture imagesof the physical environment.
 8. The system of claim 7, wherein to outputimages that include image data representative of the first physicalenvironment, the application engine is further configured to: outputimages captured by the plurality of cameras.
 9. The system of claim 7,wherein to output the plurality of images of the first user, theapplication engine is further configured to: output images of the firstuser captured by the plurality of cameras.
 10. The system of claim 1,wherein to enable communication between the first user and the seconduser, the application engine is further configured to: enable two-wayaudio and video communications between the first user and the seconduser.
 11. A method comprising: capturing, by a computing system thatincludes a head-mounted display (HMD) worn by a first user, image datarepresentative of a physical environment that includes a reflectivesurface, a plurality of objects, and the HMD; determining, by thecomputing system and based on the image data, a map of the physicalenvironment including position information about the HMD, the reflectivesurface, and each of the plurality of physical objects; enabling, by thecomputing system, communication over a network between the first userand a second user, wherein the second user is located in a secondphysical environment; determining, by the computing system and based onthe position of the HMD and the reflective surface, whether the firstuser is located in front of the reflective surface; responsive todetermining that the first user is located in front of the reflectivesurface, outputting, by the computing system and over the network, aplurality of images of the first user; and responsive to determiningthat the first user is not located in front of the reflective surface,refraining from outputting images of the first user.
 12. The method ofclaim 11, wherein the physical environment is a first physicalenvironment, wherein the HMD worn by the first user is a first HMD,wherein the reflective surface is a first reflective surface, andwherein the method further comprises: receiving, by the computingsystem, image data captured by a second HMD worn by the second user inthe second physical environment, wherein the image data captured by thesecond HMD includes images reflected by a second reflective surface inthe second physical environment, and wherein the second physicalenvironment is located a remote distance from the first physicalenvironment.
 13. The method of claim 12, further comprising: generating,by the computing system and based on the received image data, artificialreality content that includes an image representing the second user. 14.The method of claim 13, wherein outputting the plurality of images ofthe first user includes: enabling the second HMD worn by the second userin the second physical environment to present, to the second user, imagedata representative of the first physical environment.
 15. The method ofclaim 11, further comprising: rendering, by the computing system and fordisplay at the HMD, the artificial reality content overlaid on an imageof the reflective surface.
 16. The method of claim 11, whereinoutputting the plurality of images of the first user includes:outputting images that include image data representative of the firstphysical environment.
 17. The method of claim 16, wherein the computingsystem includes a plurality of cameras positioned within the physicalenvironment, and wherein capturing the image data includes: capturing,by the plurality of cameras, images of the physical environment.
 18. Themethod of claim 17, wherein outputting images that include image datarepresentative of the first physical environment includes: outputtingimages captured by the plurality of cameras.
 19. The method of claim 17,wherein outputting the plurality of images of the first user includes:outputting images of the first user captured by the plurality ofcameras.
 20. A non-transitory computer-readable medium comprisinginstructions for causing processing circuitry of an artificial realitysystem having an image capture system that includes a head-mounteddisplay (HMD) worn by a first user to perform operations comprising:capturing image data representative of a physical environment thatincludes a reflective surface, a plurality of objects, and the HMD;determining, based on the image data, a map of the physical environmentincluding position information about the HMD, the reflective surface,and each of the plurality of physical objects; enabling communicationover a network between the first user and a second user, wherein thesecond user is located in a second physical environment; determining,based on the position of the HMD and the reflective surface, whether thefirst user is located in front of the reflective surface; responsive todetermining that the first user is located in front of the reflectivesurface, outputting, over the network, a plurality of images of thefirst user; and responsive to determining that the first user is notlocated in front of the reflective surface, refraining from outputtingimages of the first user.